Skip to content

[SPARK-32381][CORE][FOLLOWUP][test-hadoop2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7#30447

Closed
sunchao wants to merge 3 commits into
apache:masterfrom
sunchao:SPARK-32381-followup
Closed

[SPARK-32381][CORE][FOLLOWUP][test-hadoop2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7#30447
sunchao wants to merge 3 commits into
apache:masterfrom
sunchao:SPARK-32381-followup

Conversation

@sunchao

@sunchao sunchao commented Nov 20, 2020

Copy link
Copy Markdown
Member

What changes were proposed in this pull request?

Revert the change in #29959 and don't remove SerializableFileStatus and SerializableBlockLocation.

Why are the changes needed?

In Hadoop 2.7 FileStatus and BlockLocation are not serializable, so we still need the two wrapper classes.

Does this PR introduce any user-facing change?

No

How was this patch tested?

N/A

@sunchao sunchao mentioned this pull request Nov 20, 2020
Closed
@github-actions github-actions Bot added the CORE label Nov 20, 2020
@dongjoon-hyun

dongjoon-hyun commented Nov 20, 2020

Copy link
Copy Markdown
Member

Thank you, @sunchao !

cc @holdenk

@sunchao sunchao changed the title [SPARK-32381][CORE][FOLLOWUP][HADOOP-2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7 [SPARK-32381][CORE][FOLLOWUP][test-hadoop2.7] Don't remove SerializableFileStatus and SerializableBlockLocation for Hadoop 2.7 Nov 20, 2020
@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36038/

holdenk
holdenk reviewed Nov 20, 2020
View reviewed changes
Comment thread core/src/main/scala/org/apache/spark/util/HadoopFSUtils.scala Outdated
holdenk
holdenk approved these changes Nov 20, 2020
View reviewed changes

@holdenk holdenk left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM pending Jenkins, one minor suggestion re: adding a comment.

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36038/

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36040/

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36040/

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Test build #131432 has finished for PR 30447 at commit c4a188c.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@sunchao

sunchao commented Nov 20, 2020

Copy link
Copy Markdown
Member Author

Cool the jenkins CI passed and the failure in github CI seems not related. I updated the PR with some comments.

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Test build #131434 has finished for PR 30447 at commit 96d0578.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36048/

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36048/

@holdenk

holdenk commented Nov 20, 2020

Copy link
Copy Markdown
Contributor

Jenkins retest this please.

@SparkQA

SparkQA commented Nov 20, 2020

Copy link
Copy Markdown

Test build #131442 has finished for PR 30447 at commit d4e542e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA

SparkQA commented Nov 21, 2020

Copy link
Copy Markdown

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36059/

@SparkQA

SparkQA commented Nov 21, 2020

Copy link
Copy Markdown

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/36059/

@SparkQA

SparkQA commented Nov 21, 2020

Copy link
Copy Markdown

Test build #131453 has finished for PR 30447 at commit d4e542e.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

dongjoon-hyun
dongjoon-hyun approved these changes Nov 21, 2020
View reviewed changes

@dongjoon-hyun dongjoon-hyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. Thanks, @sunchao and @holdenk .
This passed at #30447 (comment)
Merged to master to recover Hadoop 2.7 profile.

@sunchao sunchao deleted the SPARK-32381-followup branch November 21, 2020 02:51
@sunchao

sunchao commented Nov 21, 2020

Copy link
Copy Markdown
Member Author

Thanks @dongjoon-hyun and @holdenk !

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@holdenk holdenk holdenk approved these changes
@dongjoon-hyun dongjoon-hyun dongjoon-hyun approved these changes

Assignees

No one assigned

Labels

CORE

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@sunchao @dongjoon-hyun @SparkQA @holdenk